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ABSTRACT 

This study examines the relationship between an 
open-ended cloze test and its mult* pie-choice versions generated from 
♦hree sources: (1) interlingual learner-generated 
di stractors--distractors selected from one language group and 
administered to a different language group, (2) intralinguai 
learner- generated distractors — distractors selected fron one lanquage 
group and administered to a similar sample of the same language 
group, and (3) teacher-made distractors — distractors generated by a 
team of trained ESL/FL instructors and administered to their 
students* Open-?nded cloze tests, multiple-choice cloze formats using 
intralingual learner-generated distractors, and cloze formats 
incorporating teacher-made distractors appear somewhat similar in 
terms of their relationship to general ESL proficiency. In selecting 
a cloze passage in terms of its discourse, teachers/testers need be 
aware of and sensitive to their learners' neetfs and goals. If a 
language program involves an English for science and technology 
focus, the appropriate discourse to be taught/tested includes science 
and technology as academic subjects. If „ language program involves a 
professionally heterogeneous group of learners, a variety of 
authentic discourse types need to be tested/taught. Some cloze 
passages are more appropriate than others for language teaching, 
testing, and learning. Tests and distractors are included. 
(Author/JK) 
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correct response. 

The cloze tests were then rebuilt in a multiple-choice format 
and administered to 96 scientists studying in the Peopled Republic 
of China, Also constructed and administered were multiple-choice 
cloze tests incorporating teacher-made distractors and distractors 
from the ^hinesV* scientists 1 errors. 

It was found that there was no significant difference in the 
sample variance attributable to the types of EST discourse used. 
However, a greater similarity between the science as academic subject 
discourse and ESL reading comprehension was indicated. In additiozf, 
it appeared that multiple-choice cloze format using teacher-made or 
learner-generated distractors may be similar to open-ended cloze 

tests. 

c 

•This article is a revised version of the paper presented at the 
Fifteenth Annual TESOL convention! Detroit. 198l, under the title 
"Learner-Generated Distractors: A Synthesis of Clo2e Procedure and 
Multiple-Choice Format". 

The authors would like to thank the teaching staff at INELEC and 
GELC for their help and support. Any errors found are solely our own. 
George and Celeste Scholz are currently with the Guangzhou English 
Language Center, a UCLA China Exchange Program in the People 1 6 Republic 

of China. 



A good test should be reliable, valid and practical, A reliable test has 
stable and consist en t^cores. The validity of a test examines what a test is 
in fact measuring and how well it does so. Practicality views a test in terms 
of how easy/difficult that test is to create, administer, score and interpret. 
Working within these parameters of a good test, a program can then begin to 
invest time and money into test development. 

Apart from examining tests qua tests in the ESL/FL field, a language* test 
may be used to look at learner characteristics as well as "hypotheses about psy- 
cholinguistic and sociolinguistic factors in the performance of language based 
tasks" (Oiler 1979 :*0. If a language test can indicate and/or tap possible 
learner strategies and plausible hypotheses about the nature of language learning, 
then the test has immense instructional value. Cloze procedure as well as other 
pragmatic language- tests/tasks (eg. dictation) appear to be a satisfactory lan- 
language learning tool, as they challenge the hypothesized learner's "pragmatic 
expectancy grarimar" (Oiler 1979) • 

As a test, cloze procedure in an ESL/FL context has i>een shown to fce reliable 
within a range from .61 to .95 (Brown 1980). When correlated with various dis- 
crete-point criterion measures of English language proficiency, cloze procedure 
has yielded validity coefficients of .63 to .89, a shared variance of k6 to 
70# (Brown I98O). Apart from being a test of hypothesized overall language pro- 
ficiency, cloze procedure may also be considered a reliable and valid test of 
read^ttg^comprehension, having a validity coefficient of .60 to .70 when cor- 
relate^ with multiple-choice reading tests as a criterion measure (Oiler 1979: * 
357, Aitken 1977). 

In terms of practicality, a claze test is relatively easy to create and 
administer. A passage of appropriate content and length is selected. Allowing 
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for soae context at the beginning and end of the passage, every nth word is 
systematically deleted (usually 5th to 7th word deletions are used). There 
should be betwe m 25 to 50 deletions. The amount of time needed to administer 
a cloze test varies, but 30 minutes seems to be appropriate for a 50 item 
passage* 

One major problem of practicality remains with cloze. In comparison to the 
multiple-choice format of standardized language tests, cloze is more difficult 
to score in terois of time and word acceptability by groups of evaluators. Word 
acceptability may possibly be el; _ted as a problem by scoring only for exact 
words. (the original word deleted from the passage), but scoring for acceptable 
learner responses is fairer to the learner* given the nature of the task of com- 



pleting a cloze test. Ten to fifteen minutes per 50 item passage are, usually 
needed for correction. 

One alternative to facilitate the correction of a cloze test is to incor- 
porate s multiple-choice format in' a cloze deletion instead of a blank to be 
completed by a learner. The problem then becomes one of creating distractors. 
Conceptual problems also arise as to whether or not a multiple-choice cloze 
*ould function similarly as compared tc an nth deletion open-ended cloze. 
Would a multiple-choice cloze test be considered an integrative or discrete- 
point task? Would a multiple-choice cloze test require productive or receptive 
performance from the learner? 

In distractor creation for a multiple-choice cloze test, two studies re- 
viewed used learner-generated districtors selected on the basis of most frequent 
errors per deletion from an open-enued cloze (Brown 198O; Hinofotis and Snow 
1980). In the Brown study, distractors were selected from an intermediate-advan- 
ced pretest ESL sample of iSk. Distractors for the Hinofotis and Snow study were 
selected from a sample of 107 students entering an ESL program in the US at 
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various profici levels. In boll, bfjdies an intermediate ESI text was 
used for a cloze passage. The two studies found relatively high correlations 
between an open-ended cloze and its multiple-choice version. 

The present study attempted to exarfine the relationship between an Open- 
ended cloze test and its multiple-choice versions generated from three sources: 

1) Interlingual learner-generated distractors - dislractors selected from 
one language group and administered to a totally different language group f 

2) Intralingual learner-generated distractors - distractors selected from 
one language group and administered to a similar sample of the same 
language group, 

3) Teacher-made distractors - distractors generated by a team of trained 
ESL/FL instructors and administered to their students. 

In addition, an attempt was made to explore the relationship between two cloze 
passages (and their multiple-choice versions) of different but related scientific 
discourse: . science as academic subject and science as topic of popular interest 
(Widdowson 1976). 

Specifically, the hypotheses were: 
V 1) There is no significant difference between the concurrent validity of a 

cloze passage, its multiple-choice formats, the. criterion of ESL proficien- 
cy and its subtests. 
_ 2) There is no significant difference between cloze passages and their mul- 
tiple-choice formats with regard to the different discourse of a) science 
at academic subject and b) science as topic of popular interest. 
3) There is no significant difference between the concurrent validity of two^ 
cloze passages of different scientific discourse, their multiple-choice 
formats, the criterion of ESL profic.iency and -its subtests. 
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METHOD 
Subjects 

The subjects were derived from two sources: the Institut National d f Elec- 
tricite et Electronique (INELEC) in Boumerdes, Algeria, and the Guangzhou Eng- 
lish Language Center (GELC), a UCLA-China Exchange Program in the People f s 
Republic of China. The INELEC students were studying electrical technology 
and engineering through courses taught at the Institute in English by instruc- 
tors from the and England. Their INELEC Englisn training which prepared them 
to attend these courses was given both prior to and at the same time as the 
technical courses. These technical courses dealt with material that was equi- 
valent to the freshman year at a US university/technical institute in these 
disciplines. The students ranged in age from 17 to 2k and all had either Arabic 
or Arabic/Kabylie Berber as a first language and French as a second language. 
All of the students had received similar educational training in primary and 
secondary schools in Algeria including English language instruction. This in- 
struction was usually taught by non-native speakers through a traditional gram- 
mar orientation. Phe students could tslk about grammar but not use English for 
communicative purposes. 

The students at the Guangzhou English Language Center were scientists, the 
majority of whom were in the fields of Engineering, Physics, Chemistry and Biolo- 
gy and preparing to study in the US, England and Canada. (Other fields included 
Medicine and various Agricultural Sciences.) The great majority of these 
scientists were involved in education as l«?cturers, researchers and professor^"' 
in universities ar.fi technical institutes throughout China, while some worked in 
industry. For the most part, the students ranged in age from 37 Ao and had 
been selected from many regions of China. This was reflected in their native 
dialects: Ctuilmese, Shanghaiese, Cantonese, Fujianese, Zhejianese, Wuhanese, 



Shandorigese, Sichuanese, Jianshuese and Mandarin. While most GELC students re- 
ceived formal classroom English language instruction in secondary school and/or 
university/technical institute between twenty and thirty years ago, a small per- 
centage had never studied English formally before. Only one-fourth of all the 
students had received specialized English language training in the past six or 
seven years. Little, if any, of this English instruction had been given by 
native speakers. The remaining seventy-five per cent of the students had stu- 
died on their own using a variety of resources, including a limited number of 
textbooks and tapes available in China, radio programs and educational tele- 
vision. A very small percentage of the students could speak or read Russian, 
German or Japanese. 
Materials 

Two passages were selected, NEWTON, containing hl$ words, and ENERGY, con- 
taining MtO words. Both passages were selected using the criterion that the 
passages must be taken from texts that a native speaker of English majoring in 
science or technology would probably encounter during the freshman year at a 
US university. An ESL instructor selected the science as topic of popular in- 
terest text, ENERGY, while a science teacher selected the science as academic 
subject text, NEWTON (see Appendix A). The latter passage was selected from 
a^physics textbook and the former from a freshman, composition textbook. Every 
7th word in both passages was deleted, with one or two sentences at the beginning 
and end of every passage left intact to permit context, for a total of 50 items 
per passage. 

In order to establish the concurrent validity 0 f the NEWTON and ENERGY 
close passages and their multiple-choice formats, subtests from various stan- 
dardized ESL batteries were selected. The criterion for choosing a particular 
subtest was the reliability of the subtest to the GELC sample. The five sub- 



tests were: the Comprehensive English Language Tests of Structure and Listening 
(GELT: Structure, CL'LT: Listening), and the Test of English as a Foreign Language 
subtests of readier, vocabulary and writing (TOEFL: Reading, TOEFL: Vocabulary 
and TOEFL: Writing), The total score of these subtests was labelled GELC: Pro- 
ficiency. 
Procedure 

Four multiple-choice formats were constructed for this study, -INTER, 
WEV-INTER, -INTRA, and -TEACH (see Appendix B). l6l students at INELEC were 
administered both WEWTON and ENERGY cloze passages in July 1977. Based on raw 
scores on each cloze passage, the students were placed into three groups: high 
(top 25#), low (bottom 25#) and tnid (26-7W. From this arrangement an attempt * 
was made -to select the most frequent error in each group* However, the majori- 
ty of errors for -m item/grcup was similar. The procedure was then revised to 
select the three most frequent wrong responses across all groups, a .process simi- 
lar to the Brown * 1980) and Hinofotis and Snow (1980) studies. The three inter- 
lingual distracror*; and the exact word from the cloze passages were then put into 
* multiple-choice cloze format labelled NEWTON-INTER and ENERGY- INTER. 

The original .loze passages, NEWTON and ENERGY, and the multiple-choice for- 
mate of NEWTON- IN? MR and ENERGY- INTER were administered to 91 and 96 GELC/Chinese 
students respectively in November 1980. This group of students was called 
Cample-1. In addition, the five subtests of GELC:Proficiency were administered. 
Selection of students for each grou^ was random. Test administration procedure 
eliminated an ordering effect. NEWTON and ENERGY were scored on the acceptable 
word basis. NEWTOn- INTER and ENERGY-INTER were scored on the basis of the exact 
word contained in the multiple-choice format. 

The multiple-choice cloze passages, NEWTON-INTER and ENERGY-INTER, were item 
analyzed using th* parameters of .15 through .85 for item facility and above .25 
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eliminated. (While this procedure corrupted the 7th word deletion pattern, 
there%was no alternative. * As the distractors used in this study were learner- ' 
generated, only a very limited number were available.) Hence, the second 
multiple-choice doze format, REV NEWTON-INTER and REV ENERGY-INTER, were derived 
from the interlingual distractors contained in NEWTON-INTER and ENERGY-INTER. 

The errors generated from the original NEWTON and ENEFGY cloze passages 
administered^ to the GELC/Chinese sample were selected as intralingual distrac- 
tors on the basis of the three most frequent incorrect responses per item. In 
this fashion, the third multiple-choice format of NEWTON -INTRA and ENERGY-INTBA 
was developed. 

The fourth multiple-choice cloze format was constructed by the GELC instruc- 
tional team composed of Chinese and American ESL teachers. Working in groups, 
the teachers wrot» what they felt to be the three most effective distractors for 
each passage. In this way, NEWTON-TEACH and ENERGY-TEACH were created. 

The three multiple-choice formats of REV- INTER, -INTRA and -TEACH were then 
administered to tnree randomly selected groups at GEU in January 1981. This 
group of students was identified as Sample-2. The same subtests of GELC: Profi- 
ciency as administered to GELC Sample-1 in November 1980 were also given at that 
time. Test administration procedure eliminated an ordering effect. 
Results 

The summary test statistics o^ the criterion measures are given for the 
GELC/Chinese Saraple-1 and Sample-2 in Table 1. In terras of reliability for the 
criterion measures, estimates ranged from .52 (CELT: Listening, GELC/Chinese 
Sample-2) to .90 (TOEFL: Vocabulary and GJTC: Proficiency, GELC/Chinese Sample-2). 
The most reliable criterion measure across the GELC/Chinese samples was GELC: 
Proficiency, with a reliability estimate of .8? and .90 for GELC/Chinese 
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Sa-sple-1 and . ~0 respectively. T.ne least reliable criterion measure 

appeared to be »' !: Listening, watr. s reliability estimate o; .58 and .52 
for the Chinese nappies. TOEFL: Writing had similarly low reliabilities for 
the Chinese samples (.55 and .60). Tne reliability estimates may have been af- 
fected by- the homogeneity of the samples with regard to a specific task and/or 
the number of items in eacr. criterior. nea&ure. Tne hign reliability of the 
uELC: Proficiency total may have beer, -ue in part to the large number of items 
involved (2;55). 



Insert Table 1 about here. 



Summary te«,r statistics for JiEU'Tui*, ENERGY and related multiple-choice 
cJoze formats rrt- jiven m Table 2.- The reliability estimates for the GELC/ 
Jhmese samples oi the two cloze passages and their alternative multiple-choice 
formats revealed - fairly consistent reliability across formats which had 50 
items (.52 to .68). The highest reliability for each cloze passage was with 
the IKELEC/Algeri-u: sample (.76 - KEVTON, .70 - ENERGY) • This may have been 
due to more hetero-reneity (hence higher reliability) in the riELEC/Algerian 
sample. 

The lowest reliability estimate for each cloze passage and formats was .52 
for HEWWW-IKTS *id AS for REV E?,TJ?Gi-IMTER. The greatest disparity in re- 
liability between ;loze passages of similar format occurred with the REV NEWTON- 
INTER (.67) and ENERGY-INTER (.45). While the reliability estimates were 
lower, in general, than previous close studies, sample homogeneity may have been 
responsible. 



Insert Table 2 about here. 



To test ii tr.ere wore any significant differences between the INELEC/A1- 
gerian sample, GELC/Chinese samples on the original cloze passages and the 
criterion measures, F-ratios were used. These results may be found in Table 3^ 
It appeared that in terms of the cloze passages, the Chinese samples scored 
significantly higher than the Algerian sample (Table 3A). 

Table 3B reveals that there was no significant difference between the GELC/ 
Chinese samples, viith trie exception of vocabulary. This difference, however, 
did not af f ect tr.c- ?-ratio of GELC: Proficiency scores at any statistically 
significant level. Thus, in terms of overall proficiency, its composite mea- 
sures of structure, listening, reading comprehension arid writing, the GELC/ 
Chinr.se samples were similar. 



Insert Table 3 about here. 

To examine if there werfe any* significant differences between 1 oze pas- 
sages of similar format (eg. NEWTON-INTRA and ENERGY- INTRA) t-ratios for testing 
the difference between correlated variances were calculated. The t-tests com- 
puted indicated that there was no significant difference between scores on 
NEWTON and ENERGY , NEWTON-INTER and ENERGY-INTER, REV NEWTON-INTER and REV ENERGY- 
INTER, NEWTON-INTRA and ENERGY-INTRA and NEWTON-TEACH and ENERGY-TEACH. This 

# 

seems to indicate that there was no significant difference in sample variance 
between a text sele ted as a sample of science as academic subject and a text 
selected as a sample of science as topic of popular interest. This observation 
was supported across two distinct language samples and various cloze test 
formats. 

The validity (correlation) coefficients of the open-ended cloze and four 
multiple-choice cloze formats are given in Table k. To determine if there were 
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Table 1 



Summary Test Statistics 
for Criterion Measures 



A) GELC/Chinese Sample-1 



Test N Items Range x SD Reliability SEM 

(kr 21 ) 



CELT: Structure 


187 


75 


53-74 


66.08 


4.84 


.67 


2.77 


CELT: Listening 


187 


50 


16-41 


29.01 


5.29 


.58 


3.44. 


TOEFL: Reading 


187 


30 


6-28 


17.48 


4.76 


.70 


2.60 


TOEFL: Vocabulary 


187 


4o 


9-37 


24.83 


6.69 


.81 


2.92 


TOEFL: Writing 


187 


4o 


12-34 


25.62 


4.45 


• 55 


2.99 


GELC: Profici.-ncy 


187 


235 


109-210 163.03 


19.50 


.87 


6.97 


B) GELC/Chinese Sample-2 














CELT: Structure 


167 


75 


46-75 


66.82 


5.63 


.78 


2.64 


CELT: Listening 


167 


50 


15-46 


28.55 


4.98 


.52 


3.46 


TOEFL: Reading 


167 


30 


5-28 


18 >8 


4.97 


.73 


2.56 


TOEFL: Vocabulary 


: 167. 


4o 


12-39 


30.11 


7.66 


.90 


2.48 


TOEFL: Writing 


167 


4o 


14-35 


25.59 


4.70 


.60 


2.98 


GELC: Proficiency 


167 


235 


m-205 168.98 


21.29 


.90 


6.76 
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Table 2. 

Summary Teat Statiatice for cfoze Teata 



Cloze Teat 
Type 



Sample 



NEWTON 
ENERGY 



Cloze Teat 
Format 



N It em 8 Range 



SD Reliability SEM 



INELEC/Algerian cpeu-ended 

7th word deletion 

INELEC/Algeri nn open-ended 

?th word deletion 



161 50 5-1*5 28.31* 6.98 
161 50 10-ifO 25. a 6.29 



(KR 21 ) 



.76 3.1*2 
.70 3.k6 



NEWTON '.^C/Chineae open-ended 91 50 21-1*5 33. ^7 5.80 .68 ' 3 26 

ot^Ze-l 7th word deletion 

ENERGY GELC/Chiaeae open-ended 91 50 20-1*2 31.75 5.20 . 58 ^ ^6 

Sample-1 7th word deletion • 



NEWTON-INTER 

ENERGY-INTER 
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INTER 



GELC/Chinese 
Sample-1 

GELC/Chinese 
Sample-1 



multiple-choice 96 
Interlingual distractors 

multiple-choice 96 
Interlingual distractors 



50 2>^3 35.0^ k.70 .5^ 3.20 

SO 2*f-**6 35.^7 k.63 .53 3.18 



?SL^ EWT0N " GELC/Chineee multiple-choice 57 .20 5-20 12.86 3.51* .67 1.18* 

iWlhH Sample -2 Interlingual distractors 

revised through item 
. analysis 

^^ENER<r/- GELC/Chine<ye multiple-choice 57 20 6-17 11.^2 2.99 
Sample-2 Interlingual distractors 



revised through item 
analysis 



■EWTON-INTOA GELC/Chinese multiple-choice 56 50 19-M 30.96 5.56 

Sample-2 Intralingual distractora 

ENERGY- INTRA GELC/Chinese multiple-choice 

Saaple-2 Intralingual distractors 



.<f8 



2.16 



.63 3.38 

56 50 19-<*2 30.1^ 5.60 .63 3.<K> 



riErfTOM-TFACH 



.■MSfi5Y-TEACH 
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GELC/Chineae 
S. ..ple-2 

GELC/Chinese 
Sample -2 



multiple-choice 54 
teacher-made distractors 



50 2W*2 33.63 ^.76 



multiple-choice 
tuacaer-made diScr.nctoru 



?*» 50 26-45 36.02 'f.67 



• 52 
.55 



3.30 
3.13 



A) 



Table 3 

F -Ratio to' Test for Siffiificant 
Differences in Population Variance 



Test 



Sample 



df 



SS 



NEWTON 



INELEC/Algerian 160 1373^8 28.3^ 

GELC/Chinese 90 10^990 33.47 

Sample-1 



1.36 (.05) 



ENERGY 



INELEC/Algerian 160 

GELC/Chinese 90 
Sample-1 



108671 25.21 
94731 -31.75 



1.55 (.01) 



B) 



CELT: 


Structure 


GELC/Chinese 
Sample-1 


186 


820907 


66.08 






GELC/Chinese f 
Sample- 2 


166 


750911 


66.82 


CELT: 


Listening 


GELC/Chinese 
Sample-1 


186 


I62587 


29.01 






GELC/Chinese 
Sample-2 


166 


H*0252 


28.33 


TOEFL; 


Reading 


GELC/Chinese 
Sample-1 


186 


61320 


17.1+8 






GELC/Chinese 
Sample- 2 


166 


58673 


18.08 

# 


TOEFL: 


Vocabulary 


GELC/Chinese 
Sample-1 


186 


123658 


24.83 






GELC/Chinese 
Sample-2 


166 


161118 


30.11 


TOEFL: 


Writing 


GELC/Chinese 
Sample-1 


186 


126*33 


25.62 






GELC/Chinese 


166 


II3OO5 


25.59 


GELC: 


y 


GELC/Chinese 
Sample-1 


186 


5141065 


163.03 






GELC/Chinese 
Sample-2 


166 


481*3611 


168.98 



1.02 



1.03 



1.07 



1.1*6 (.01) 



1.00 



1.08 



*p>.05 unless otherwise indicated. 
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any differences between the validity coefficients of similar formats of 
NEWTON and ENERGY, Hotelling t values were calculated. Reviewing the concur- 
rent validity coefficients in Table U sh^ws moderately strong correlations of 
both close passages in an open-ended format with GELC: Proficiency (.67 - NEWTON, 
.Sk - ENERGY) and reading (.73 - NEVTON, .61 - ENERGY). As well, there were 
similar validity coefficients of both cloze passages in a teacher-made multi- 
ple-choice format with overall GELC: Proficiency (.62 - NEWTON-TEACH, .?k - 
ENERGY-TEACH). NEWTON (open-ended) correlated significantly higher with the 
criterion measure of reading comprehension than ENERGY (open-ended) f the dif- 
ference in shared variance being l6#. In general, however, 'the -INTER, -INTRA 
end -TEACH multiple-choice formats of J^ERGY correlated significantly higher with 
the crfterion measures of reading comprehension and/or GELC: Proficiency than 
NEWTON passages of the same formats. In addition, the cloze passages revised 
through item analysis generally had the lowest validity coefficients of all 
cloze formats involved in this study. 



Insert Table k about here. 



To examine if there were any significant dif|*rences between a cloze pas- 
sage, its alternative multiple-choice formats and tho criterion measures, 
Fisher 1 s z transformation was used. The results of those measures in which sig- 
nificant differences were found are contained in Table 5. 

There appeared to be a significant difference in the concurrent validity 
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Table 4 

Concurrent Validity Coefficients 
A) QELC/Chinese Sample-1 (N=9l) 



NEWTON ENERGY 





r 


Variance 
(r 2 ) 


r 


Variance 

(r 2 ) 


Hotelling t 


P* 


* difference 
in variance 


ENERGY 


.60 


36* 












CELT: Structure 


.39 


15* 


.40 


16* 


.12 






CELT: Listening 


•51 


26* 


.43 


18* 


1.49 






TOEFL: Reading 


•73 


53* 


.61 


37* 


1.94 


(.03) 


16* 


TOEFL: Vocabulary 


.42 


18* 


.42 


18* 


0 . 






TOEFL: Writing 


.51 


26* 


.52 


27* 


.13 






GELT: Proficiency 


.67 


45* 


.64 


4i* 


.48 







B) QELC/Chinese Sample-1 (N=96) 



NEWTON-INTER ENERGY- INTER 

*"r Variance ~~r Ve-iance Hotelling t p* * difference 
(r2) ( r 2) 1 in variance 



ENERGY -INTER 


• 50 


25* 












CELT: Structure 


.39 


15* 


.39 


15* 


0 






CELT: Listening 


• 31 


10* 


.35 


12* 


.42 






TOEFL: Reading 


.1*2 


18* 


• 57 


32* 


1.79 


(.04) 


14* 


TOEFL: Vocabulary 


-21 


4* 


.40 


16* 


2.00 


(.02) 


12* 


TOEFL: Writing 


.45 


20* 


.53 


28* 


.94 






GELC: Proficiency 


.^5 


20* 


.58 


34* 


1.58 







1* 



*p?.05 unless otherwise indicated. 
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Table k (continued' 



C) GELC/Chinese Sample-2 (N=57) 



REV NEWTON-INTER REV ENERGY- INTER 





r Var 


iance 
Cr2) 


r 


Variance 

(r2) 


Hotelling t 


P* 


% difference 
in variance 


REV ENERGY-INTER 


*39 


15* 












CELT: Structure 


.29 


, 8* 


.28 


8* 


7 






CELT: Listening 


.29 


8* 


.08' 


,6* 


1M 


— 


— 


TOEFL: Reading 


.56 


31* 


•39 


15* 


1M 






TOEFL: Vocabulary. 16 


3% 


•31 


10* 


1.05 






TOEFL: Writing 


.36 


13* 




10* 


.29 






GELC: Proficiency. k2 


18* 


•38 


lk% 


• 30 


— 


— 


D) GELC/Chinese 


Sample-2 


(N=5*0 












NEWTON-INTRA 


ENERGY-INTRA 










r Variance 

(r2) 


r 


Variance 
(r2) 


Hotelling t 




% difference 
in variance 


ENERGY- INTRA 


.63 


U<# 












CELT: Structure 


' -33 


11* 


A3 


18* 


M * 






CELT: Listening 


M 


17% 




2k% 


.79 






TOEFL: Reading 


.53 


28* 


.6k 


kl% 


1.24 






TOEFL: Vocabulary .35 


12* 


,h2 


18* 


.66 






TOEFL:' Writing 


.kz 


18* 


.55 


30* 


1.33 






QELC: Proficiency .^9 


2k% 


.66 


kk% 


1.93 


(•03) 


20# 



*p >.05 unless otherwise indicated. 
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Table k (continued) 
E) GELC/Chinese Sample-2 (N=5<0 



NEWTON-TEACH ENERGY-TEACH 

r Variance r Variance Hotelling t p* % difference 
(r2) (r2) in variance 



ENERGY-TEACH 


.71 


J% 












CELT: Structure 


.35 


1236 


.48 


23* ' 


1.39 






CELT: Listening 


.44 


19* 


.48 


23* 


.43 






TOEFL: Reading* 


.55 


30* 


.69 


48* 


1.83 


(.03) 


18* 


TOEFL: Vocabulary 


.53 


28* 


.66 


44* 


1.63 






TOEFL: Writing 


.5** 


29* 


.59 


35* 


.59 






GELC: Proficiency 


.62 


38* 


.74 


55* 


1.71 


( .04) 


17* 



*p > # 05 unless otherwise indicated. 
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11. 

coefficients > ^ifti<*j ns acac«-r: . ::,ject close package, NKWTOU, and its 
multiple-cnoice formats with respect to reading comprehension. In terms of 
possible overall ESL proficiency as measured by GELC: Proficiency, the picture 
is less clear, as it appears that the open-ended cloze format, its multiple- 
choice interlingual ai:d its multiple-choice teacher-made distractor formats 
had similar concurrent validity (r^ = 2k to ^5 

Focussing on ENERGY, there appeared to be no si^ ficant difference bet- 
ween the validity coefficients of this science as topic of popular interest 
cloze and its multiple-choice formats with GELC: Proficiency, provided that 
items were not discarded due to item analysis. In terms of the various multi- 
ple-choice formats, intralingual (-INTRA) and teacher-made (-TEACH) distrjictors 
appeared to be more similar than interlingual distraqtors. The difference bet- 
ween validity coefficients and the criterion of reading comprehension was negli- 
gible, with the ubiquitous exception of REV ENERGY-INTER • One interesting 
anomaly was that the teacher-made multiple-choice distractor format of the 
science as topic of popular interest cloze, ENERGY-TEACH, correlated signifi- 
cantly higher (r = .66) with the vocabulary criterion than any other format of 
the same passage. As well, there was a significant difference of GELC/Chinese 
Caaple-2 in vocabulary as compared to GELC/Chinese Sample-1 (see Table 3). 
Terhaps this particular cloze passage and distractor type served more as a mea- 
sure of a receptive vocabulary task than as afc integrated task of reading or 
overall ESL proficiency. 



Insert Table 5 about here. 



Item facility and discrimination indices are reported in Table 6. The 
item analysis for each cloze passage and respective multiple-choice formats in- 
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Table 5 

4 Fisher's z Transformation of Cloze Test 
Validity Coefficients with Criterion Measures 
' (z values above diagonal; p* below diagonal) 

NEWTON - Science as Academic Subject 
A. 1) TOEFL: Reading 



— r (t 2 ) 


NEWTON 
.73 (53%) 


NEWTON-INTER 
.k2 (18%) 


REV NEWTON- 
INTER 

.56 (3156) 


NEWTON-INTRA 
.53 (28%) 


NEWTON-TEACH 
.55 (30*) 


NEWTON • 




3.23 


1.71 


1.95 


1.72 


NEWTON-INTER 


(.0001) 




1.08 


.83 


1.02 


REV NEWTON-INTER 


(.0*0 






.22 


.Ok 


NBWTON-INTRA 


(.03) 








.18 


NEWTON-TEACH 


(.Ok) 











-A. 2) TOEFL: Vocabulary 



NEWTON NEWTON-INTER REV NEWTON- NEWTON-INTRA NEWTON-TEACH 

INTER 



r (r 2 ) 


.kZ (18%) 


.21 (k%) 


.16 (3%) 


35 (12%) 


.53 (28%) 


NEWTON 




1.58 


l.£6 


.kS 


.81 


NEWTON-INTER 






1.30 


.88 


2. 16 


REV NEWTCN -INTER 


(.05) 






1.06 


2.20 


NEWTON-INTRA 










1.15 


NEWTON-TEACH 




(.02) 


(.01) 







*p > .05 unless otherwise indicated. 



Table 5 (continued) 



NEWTON - Science as Academic Subject 
A. 3) GELC: Proficiency 



r (r 2 ) 



NEWTON 

.67 (490 



NEWTON-INTER REV NEWTON 
INTER 



.45 (20#) 



NEWTON -INTRA NEWTON-TEACH 
.42 (18#) .49 (2435) .62 (38#) 



NEWTON 

NEWTON-INTER (.01) 
REV NEWTON-INTER ( .02) 
NEWTON-INTRA 
NEWTON-TEACH 



2.19 



2.10 
.22 



1.58 
1.60 
.46 



.49 
1.38 
1.42 

.96 



ENERGY 2 Science as Topic of Popular Interest 
B. I) TOEFL: Reading 



ENERGY ENERGY-INTER REV ENERGY- ENERGY-INTRA ENERGY-TEACH 

INTER 



r (r 2 ) 


.61 (3750 


.57 (32#) 


.39 (1550 


.64 (4l#) 


.69 (48#) 


ENERGY 




.41 


1.72 


.28 


.79 


ENERGY-INTER 


(.08) 




1.38 


.64 


1.15 


REV ENERGY-INTER 


(.04) 






1.79 


2.23 


ENERGY-INTRA 






(.04) 




.46 


ENERGY-TEACH 






(.01) 







*p > .05 unless otherwise indicated. 



Table 5 (continued) 



B. 2) TOEFL: Vocabulary 



ENERGY 



r (r ) .42 (18*) 



ENERGY-INTER REV ENERGY- ENERGY-INTRA ENERGY-TEACH 
INTER 



.to (l6*) 



.31 (10*) .to (18*) 



.66 (44*) 



ENERGY 
ENERGY-INTER 
REV ENERGY-INTER 
ENERGY-INTRA 
ENERGY-TEACH 



(.03) 



.16 



(.02) 



.73 
.60 



(.01) 



0 

.14 

.66 

(.04) 



1.96 
2.12 
2.42 
1.76 



B. 3) GELC: Proficiency 
ENERGY 



ENERGY-INTER REV ENERGY- ENERGY-INTRA ENERGY-TEACH 
INTER 



r (r 2 ) 


.64 (41*) 


.58 (34?6) 


.38 (14*) 


.66 (hk%) 


.74 (55*) 


ENERGY 




•65 


2.07 


.20 


1.09 


ENERGY-INTER 






1.53 


2.28 


I.65 


REV ENERGY-INTER 


(.02) 






2.03 


2.81 


ENERGY-INTRA 




(.01) 


(.02) 




.80 


ENERGY-TEACH 




(.05) 


(.003) 







*P y «05 unless, otherwise indicated. 
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12. 

dictates that both of the original cloze passages wer' easier for the GELC/ 
Chineae Sample than the INELEC/Algerian sample- That these differences were 
significant are supported bv the F-ratios in fable 3. In general, the mean item 
facility figures indicate that both cloze passages and formats were relatively 
easy for the Chinese sample, with a range of .61 to .72. The mean item discrimi- 
nation figures for all samples, cloze passages and formats are low (.21 to .*tl) t 
indicating that the cloze passages were unable to finely discriminate among 

the subjects in each sample- This would probably indicate a high homogeneity 
% 

of each sample wibh respect to language ability- The fairly low reliability 
estimates in Table 2 tend to Support this- 

As an item analysis is a tool used to improve a multiple-choice testing 
format by eliminating or revising items oi extremely high/low facility and low 
discrimination, an exploration of effective items in both cloze passages and 
multiple-choice formats was conducted. Due to the high interdependence of cloze 
items in an open-ended format, however, this aspect of the study can only be. 
speculative, as the* ac f ual application of item analysis may not be conceptually 
sound • 

As indicated in Table 6, computing the revised mean item facility and 
revised mean item discrimination yielded a movement towards the ideal item fa- 
cility level of «5 and an increase of the discrimination- The number of effec- 
tive items (as determined by the parameters of item analysis) in the open-ended 
cloze passages were 25 for NEWTON and 30 for ENERGY, which constituted 50# and 
60# of the original 50 items per respective passage- High commonality of ef- 
fective items was found between the open-ended cloze passages, INELEC/Algerian 
and GELC/Chinese samples. For example, with 2 r ) of the original items determined 
to be effective in the GELC/Chinese sample of NEWTON, 22 of the effective items in 
the INELEC/Algeriri-i sample were similar fci a commonality of 88# (22/25), 



13. 

With the exception of the intralingual detractors (-INTRA) and NEWTON - 
TEACH, the multiple-choice formats had a rruch lower effective item commonality 
with their original cloze passages (33 to k7%) . It seems possible that multiple- 
choice cloze formats were using somewnat different items in establishing taeir 
reliability and concurrent validity with criterion measures. However, surface 
difference in item similarity and obvious difference in testing format appeared 
to be statistically insignificant when the correlation coefficients with the 
criterion measure of GELC: Proficiency were reviewed in Table 5. One* anomaly 
in the validity coefficients was the revised interlingual multiple-choice format 
(&EV -INTER) of both cloze passages. Not only was the item commonality with the 
original cloze passage the loijgst of all formats, but the effect of contextual 
constraints by mechanical deletions was totally eroded. 



.Insert Table 6 about here 



Discussion 

While there was no significant difference between the variance of two dif- 
ferent cloze passages having similar format, there appeared to be significant 
differences between cloze passages in relation to the criterion of reading com- 
prehension. Wheroas NEWTON (oppn-ended) f involving the disease of science 
as academic subject correlated higher with reading than the science as topic of 
popular interest passage ( ENERGY) , the exact opposite was true when a multiple- 
choice ;ormat was used. One explanation may be that the science as academic 
subject cloze passage (NEWTON) correlated with the reading criterion because the 
discourse of the particular passage was similar to the discourse of the passages 
involved in the reading criterion measure. That is say, the discourse of 
Science as academic subject may be more similar to the discourse^of ESL as 
academic subject trum the discourse of science as topic of popular interest. 
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Table 6 

Item Facility and Discrimination 
for NEWTON and ENERGY Cloze Passages 
(see Appendix A & B) 



A) NEWTON 


NEWTON 


NEWTON 


NEWTON-INTER 


REV NEWTON - 
INTER 


NEWTON-INTRA 


NEWTON-TEACH 


Sample: 


GELC/Ch 


INELEC/Alg 


'GELC/Ch 


GELC/Ch 


GELC/Ch 


GELC/Ch 


Mean Item Facility: 


.66 


.55 


.70 


.6k 


.61 


.66 


Mean It«a 
Discrimination: 


.27 


.if] 


.21 




.28 


.23 


Revised Mean 
Item Facility: 


.59 


.50 


.65 


.60 


.55 


.60 


Revised Mean 

Item Discrimination: 


.HI 






OA 


• 30 




Items Remaining: 


25 


29 


20 


ik 






Item Commonality of 
Effective Items with 
NEWTON r # 




22 

(88*) 


11 : 


8 

(32*) 


Id. 

(64*) 


16 

(64*) 



B) ENERGY 



Sample: 



ENERGY ENERGY ENERGY-INTER 
GELC/Ch INELK/Alg — GELC/Ch 



REV ENERGY- 
INTER 



GELC/Ch 



ENERGY-INTRA ENERGY-TEACI 
GELC/Ch 



.72 
.22 



Mean Item Facility: 

Mean*Item 
Discrimination: 



.64 
.2? 



.47 
.52 



.70 
.22 



.56 
.3^ 



.60 
.28 



1 



Revised Mean 
Item Facility: 

Revised Mean 

Item Discrimination: 



.58 
.37 



.51 
.40 



.62 
.37 



.61 

.42 



.57 
.38 



.62 
.39 



Items Remaining: 


30 


32 


20 


15 


32 


19 


Item Commonality of 














Effective Items with 




22 


L4 


10 


22 


19 


ENERGY:* 




(73%) 


(47*) 


(33*) 


(73*) 


(40*) 



*as determined by item analysis. 
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Science and ESL as academic subjects may share the same discourse char- 
acteristics of being "typically metalinguistic, didactic, explicit" (Widdovson 
1976*266). The discourse of science as topic of popular interest on the other 
hand varies "considerably. . .depending on the readersrup aimed ?t" (Widdowson 
1976:266) • Moreover, a popular science text may be too heavily culture-bound 
aa it translates information from science that relies on the target language 
culture, "making appeal, and making concession, to social beliefs, attitudes, 
views of the world" (Widdowson 1979:52). 

When a multiple-choice format is introduced, the culture bound and stylis- 
tic aspect of the science as topic of popular interest is reduced, making the 
passage more manageable for the learner, though not necessarily as a reading 
processing task but perhaps a vocabulary task. As well, the explicit and di- 
dactic features of a science as academic subject are increased to the point 
where the test itself loses its strength as a task of reading comprehension. 

Reviewing distractors types, intralingual distractors may be related more 
closely to teacher-made distractors than to an open-ended format. This rela- 
tionship appears to be especially ►rue when both teachers and students are. in 
the sane context, ie. language program. Similarities between multiple-choice 
formats, discrete-point in nature, are greater than ) ween integrative and 
discrete-point formats. Hence, multiple-choice formats are somewhat different 
than integrative open-ended cloze formats. 

These differences, however, do not necessarily obviate the intralingual 
or teacher-made multiple-choice cloze fo.Tiats 1 membership to integrative tasks. 
The differences between a multiple-choice and open-ended cloze format may be 
due to a receptive/productive demand. A multiple-choice clofce task may be 
receptive i*i the sense that a learner need only identify the correct answers, 
While an open-ended cloze task requires written production of \ the cornet 



answer or a semantically acceptable equivalent. Moreover, what claims could 
a multiple-choice cloze format make to being discrete-point in nature when the 

A 

items are selected on the basis of nth word deletion rather than an a priori 
item selection based on syntax or ! ixis? 
Conclusion g 

Responding to the initial hypotheses in this paper, the results of this 
study indicate that open-ended cloze tests, multiple-choice cloze formats using 
intralingual learner-generated dis tractors and cloze formats incorporating 
teacher-made distractors appear somewhat similar in terms of their relationship 
to general ESL proficiency. Based on a sample of science students and scientist 
it appears that an open-ended cloze containing the discourse of science as aca- 
demic subject remains a more valid measure of ESL reading comprehension than 
its multiple-choice counterparts. On the other hand, a cloze passage represent- 
ing the discourse of science as Y a topic of popular interest appears to be of 
the same validity in terms of ESL reading comprehension as its multiple-choice 
formats employing intralingual or teacher-made distractors. 

While there is no significant difference of population variance between 
the discourse of science as academic subject and science as topic of popu- 
lar interest passages, the former seems more closely related to ESL reading com- 
prehension than the latter. A multiple-choice format proceeds to make opposite 
claims — closely relating a science as topic of popular interest discourse to 
ESL proficiency, reading and vocabulary. It appears that across discourse types 
cloze procedure and its multiple-choice varieties make competing claims. How- 
ever, in the testing of a given discourse, an open-ended cloze procedure might 
also be accomodated by both intralingual learner-generated and teacher-made 
distractors. This may serve to alleviate the problem of scoring cloze. Yet, 
instructionally, open-end^d cloze remains stronger due to its ability of ex- 
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plotting doubt and choice, two crucial elements in the development of communi- 
cative coibm*toS£7 

**~ln selecting a proper cloze passage in terras of its discourse, teachers/ 
testers need to be aware of and professionally sensitive to their learners' 
needs and goals. If a language program involved an English for Science and 
Technology (EST) focus, the appropriate discourse to be taught/tested would 
include science and technology as academic subjects. If a language program is 
involved with a professionally heterogeneous group of learners, then perhaps 
a variety of authentic discourse types would have to be tested/taught. Some 
cloze passages may be more appropriate than others with respect to language 
teaching, testing and learning. 
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Appendix A ' 
Cloze: Newton 

(Science as Academic Subject) 

Newton assumed that gravitational force depended on the masses of the attract- 
ing bodies as well as inversely on the square of their distance of separation. He 
then generalized his concept of gravitational attraction into a law of universal 
gravitation. He thought that all .bodies, no (1) (matter) « where they were located, 
exerted forces (2) (of ) * gravitational attraction upon one -another. In (3) (order? 
to discover the exact nature of (4) ( this) 'attractive force, he had to consider 1 
(5) (bodies)* of various different masses- at significantly (6) (different) distances 
from one another. Me could (7) (not)* change the distance between the center (8) (of)* 
the earth and a body on (9) (the)* earth very appreciably, however. It was (10) (forT 
this reason that he first compared (11) (the) motion of the moon and a (12) (body)* 
on earth. The force between different (13) (macroscopic)* bodies on the earth'waa so 
(14) (small) that it was not detected in (13) (Newton ' s)* time. Newton apparently 
realized that this force (16) (was) small and easily masked by frictional (17) (or) 
other fprces. Hence, he focused his (l8) (attention) on- the motion of the planets 
.(19) (in) * an attempt to confirm his ideas. 

(20) (The) *earliest scientific attempts to unde. ;.and the (21) ( solar)* system 
were madefy the -Greeks. (22)00 detailed description of the conclusions of 
(23) 'Greek) astronomers was given by Ptolemy. His (24) (system) is known as the 
Ptolemaic, or ( 25) ( geocentric) theory. It assumes that the earth (26) (is)* sta- 
tionary at the center of the (27) (universe) , with the sun, moon, planets~and 
(28) (stars) all revolving .about the earth in (29) (complex) orbits. This theory 
was accepted for (JO) Calmost) *fifteen centuries and greatly influenced philosophy 
(3D (and) literature as well as science. However,- (32) (the)* theory was quite- 
complex and could (33) (not) quantitatively account for an increasing number • 
(3*0 iofO'observations. In the sixteenth century Copernicus (35) (suggested)* that" 
a simpler description of celestial (36) (motions)« could be given by assuming that" 
(37) (the)* sun war, at rest at the (38) (center)* of the universe. "In the Coperni- ' 
can (39) (01O heliocentric theory, the earth was a (40) (planet)* rotating on its' 
axis and revolving (4i) (around) * the sun, and the other planets (l»2) (had) similar 
motions. 

The growing cqntroversy over (43) (the) two theories stimulated astronomers 
to obtain (44) (more) accurate observational data. Such data were (45) (compi led) 

by Tycho Brahe, who was the (46) (last) great astronomer to make observations 

without (47) (the) use of a telescope. His data (48) (on£ planetary motions were 
analyzed and interpreted (49) (for) about twenty years by Johannes Kepler, (50) (who) 

had been- Brahe' s assistant. Kepler found important regularities in the motion 

of the. planets. These regularities .are known as Kepler's three laws of planetary . 
motion. * J 

*items ineffective through item analysis 

(see Table 6) 

Cloze: Energ y 

(Science as Topic o! Popular Interest) 

Just after World War II, uranium seemed to many to be just another fuel that 
could be exploded in anger or consumed more slowly in calculation - doing more da- 
mage than other fuels in the first case, but more good in the second. 

Unfortunately, uranium is more than just another fuel - and worse. 

What the general public didn't realize, (1) UO first, was that the atomic 
bomb, (2) (jrfhen) exploded, did not do all its (3) ^mage)* at the moment of explo- 
sion. When (4) Uranium) undergoes fission, the ash it produces (5) (is) atoms that 
are fiercely radioactive, and (6) of) these spread out over the lanT~(7) (they) 

will carry a dead! mess that wowid TS Rlast) for decades. JLL 

i 
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Uranium undergoing controlled fission (9) Cproduced) useful energy for man- 
kind - but the (10) (radioactive) ash is'still produced; it is (11) (still) there. 
It must not be allowed (12) (to)* escape into the environment, either before 
"(13) ( disposal y or after. This aspect of fission .(1*0 ( energy ) makes it a most 
uneasy Substitute (15) (for) the fossil fuels, coal and oil. 

(16) (In) fact, l we look toward the (17) ( future) , we find that coal, o^l 
and (18) (uranium ) are alike in this: we can (19) ( depend ) on none of them for 
the (20) (21st )*century, The oil will be* for (21) (the)* most part, gone by then. 
The (22) ( coal) , in the quantities needed, will destroy (23) ( the) land in the 
course of being (24) (dug) out, ,and will pollute the air (25) (in)* the -course of 
being burned. As (26) (for)* uranium, that will bave within it, (27) (always) * 
the fearsome danger of the radioactive (28) ( contaminationJ* of the world. 

Yet we must (29) (have)* energy if * civilization is to survive. (30) (Where ) 
will it come from, if oil, (31) (coal)* and Uranium are eliminated as possible 
(32) (sources) ?* 

One thing to* remember is that (33) (there)* are many sources of energy that 
(3*0 (we) do use, have used in the (35) (past) , and can use more of in (3o) (the)* 
future. There is still the energy (37) (of) * the wind, of running water, of 
(38) (the)* tides, of the Earth's internal heat, 

(39) (fcven) all of these together may not (40) (be) enough to supply all of 
mankind* 3 (4l) (energy) needs. Properly exploited, however, they can (42) (offer) 
man a large percentage of the (43) ( energy) *he ineeds, and can relieve the 
(44) (weight) of our dependence on oil, coal (45) (and)* uranium, until the really 
suitable source (46) (is) developed. 

Each of these sources', moreover, (47) (is)> inexhaustible. Wind and running 
water will (48) (last) as long as the sun -shines, (49) ( the) tides will be there 
as long (50) (as/ the Earth turns', and the Earth will have internal heat as long 
as it exists. All that energy is there and is being expended; it is only necessary 
to use it,- rather than letting it go to waste. 

♦items ineffective through item analysis 
(see \able 6) 

I 
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